
pip install yellowbrick

Requirement already satisfied: yellowbrick in c:\users\ak153\anaconda3\lib\site-packages (1.5)
Requirement already satisfied: matplotlib!=3.0.0,>=2.0.2 in c:\users\ak153\anaconda3\lib\site-packages (from yellowbrick) (3.3.4)
Requirement already satisfied: scikit-learn>=1.0.0 in c:\users\ak153\anaconda3\lib\site-packages (from yellowbrick) (1.3.0)
Requirement already satisfied: scipy>=1.0.0 in c:\users\ak153\anaconda3\lib\site-packages (from yellowbrick) (1.6.2)
Requirement already satisfied: cycler>=0.10.0 in c:\users\ak153\anaconda3\lib\site-packages (from yellowbrick) (0.10.0)
Requirement already satisfied: numpy>=1.16.0 in c:\users\ak153\anaconda3\lib\site-packages (from yellowbrick) (1.20.1)
Requirement already satisfied: six in c:\users\ak153\anaconda3\lib\site-packages (from cycler>=0.10.0->yellowbrick) (1.15.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in c:\users\ak153\anaconda3\lib\site-packages (from matplotlib!=3.0.0,>=2.0.2->yellowbrick) (2.4.7)
Requirement already satisfied: python-dateutil>=2.1 in c:\users\ak153\anaconda3\lib\site-packages (from matplotlib!=3.0.0,>=2.0.2->yellowbrick) (2.8.1)
Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\ak153\anaconda3\lib\site-packages (from matplotlib!=3.0.0,>=2.0.2->yellowbrick) (1.3.1)
Requirement already satisfied: pillow>=6.2.0 in c:\users\ak153\anaconda3\lib\site-packages (from matplotlib!=3.0.0,>=2.0.2->yellowbrick) (8.2.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in c:\users\ak153\anaconda3\lib\site-packages (from scikit-learn>=1.0.0->yellowbrick) (2.1.0)
Requirement already satisfied: joblib>=1.1.1 in c:\users\ak153\anaconda3\lib\site-packages (from scikit-learn>=1.0.0->yellowbrick) (1.3.1)
Note: you may need to restart the kernel to use updated packages.


# Libraries to help with reading and manipulating data
import numpy as np
import pandas as pd

# Libraries to help with data visualization
import matplotlib.pyplot as plt
import seaborn as sns

# to scale the data using z-score
from sklearn.preprocessing import StandardScaler

# to compute distances
from scipy.spatial.distance import cdist, pdist

# to perform k-means clustering and compute silhouette scores
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score

# to perform hierarchical clustering, compute cophenetic correlation, and create dendrograms
from sklearn.cluster import AgglomerativeClustering
from scipy.cluster.hierarchy import dendrogram, linkage, cophenet

# to perform PCA
from sklearn.decomposition import PCA

# to visualize the elbow curve and silhouette scores
from yellowbrick.cluster import KElbowVisualizer, SilhouetteVisualizer

#format numeric data for easier readability
pd.set_option(
    "display.float_format", lambda x: "%.2f" % x
)  # to display numbers rounded off to 2 decimal places

# Removes the limit for the number of displayed columns
pd.set_option("display.max_columns", None)
# Sets the limit for the number of displayed rows
pd.set_option("display.max_rows", 200)

# To supress warnings
import warnings
warnings.filterwarnings("ignore")


# Importing andreading csv file into a pandas Dataframe
data = pd.read_csv('C:/Users/AK153/GREAT_LEARNING/Projects/MODULE 7 UNSUPERVISED LEARNING- TRADE&AHEAD/stock_data.csv')
# copying data to another varaible to preserve original data
df = data.copy()


#Checking Shape of the data
print(f"There are {df.shape[0]} rows and {df.shape[1]} columns.")

There are 340 rows and 15 columns.


# print a sample of 10 rows randomly selected from the training data
df.sample(n=10,random_state=1)


# Checking The Column Names And Datatype
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 340 entries, 0 to 339
Data columns (total 15 columns):
 #   Column                        Non-Null Count  Dtype  
---  ------                        --------------  -----  
 0   Ticker Symbol                 340 non-null    object 
 1   Security                      340 non-null    object 
 2   GICS Sector                   340 non-null    object 
 3   GICS Sub Industry             340 non-null    object 
 4   Current Price                 340 non-null    float64
 5   Price Change                  340 non-null    float64
 6   Volatility                    340 non-null    float64
 7   ROE                           340 non-null    int64  
 8   Cash Ratio                    340 non-null    int64  
 9   Net Cash Flow                 340 non-null    int64  
 10  Net Income                    340 non-null    int64  
 11  Earnings Per Share            340 non-null    float64
 12  Estimated Shares Outstanding  340 non-null    float64
 13  P/E Ratio                     340 non-null    float64
 14  P/B Ratio                     340 non-null    float64
dtypes: float64(7), int64(4), object(4)
memory usage: 40.0+ KB


# Checking for Duplicate Values
df.duplicated().sum()

0


# Checking For Misiing Values
df.isnull().sum()

Ticker Symbol                   0
Security                        0
GICS Sector                     0
GICS Sub Industry               0
Current Price                   0
Price Change                    0
Volatility                      0
ROE                             0
Cash Ratio                      0
Net Cash Flow                   0
Net Income                      0
Earnings Per Share              0
Estimated Shares Outstanding    0
P/E Ratio                       0
P/B Ratio                       0
dtype: int64


# Convert all columns with dtype object into category
for col in df.columns[df.dtypes=='object']:
    df[col]=df[col].astype('category')


# dropping the ticker symbol column, as it does not provide any information
df.drop("Ticker Symbol",axis=1,inplace=True)


# Confirm New Dataset
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 340 entries, 0 to 339
Data columns (total 14 columns):
 #   Column                        Non-Null Count  Dtype   
---  ------                        --------------  -----   
 0   Security                      340 non-null    category
 1   GICS Sector                   340 non-null    category
 2   GICS Sub Industry             340 non-null    category
 3   Current Price                 340 non-null    float64 
 4   Price Change                  340 non-null    float64 
 5   Volatility                    340 non-null    float64 
 6   ROE                           340 non-null    int64   
 7   Cash Ratio                    340 non-null    int64   
 8   Net Cash Flow                 340 non-null    int64   
 9   Net Income                    340 non-null    int64   
 10  Earnings Per Share            340 non-null    float64 
 11  Estimated Shares Outstanding  340 non-null    float64 
 12  P/E Ratio                     340 non-null    float64 
 13  P/B Ratio                     340 non-null    float64 
dtypes: category(3), float64(7), int64(4)
memory usage: 46.7 KB


df.describe(include="all").T


# function to plot a boxplot and a histogram along the same scale.
def histogram_boxplot(data, feature, figsize=(15, 10), kde=False, bins=None):
    """
    Boxplot and histogram combined

    data: dataframe
    feature: dataframe column
    figsize: size of figure (default (15,10))
    kde: whether to show the density curve (default False)
    bins: number of bins for histogram (default None)
    """
    f2, (ax_box2, ax_hist2) = plt.subplots(
        nrows=2,  # Number of rows of the subplot grid= 2
        sharex=True,  # x-axis will be shared among all subplots
        gridspec_kw={"height_ratios": (0.25, 0.75)},
        figsize=figsize,
    )  # creating the 2 subplots
    sns.boxplot(
        data=data, x=feature, ax=ax_box2, showmeans=True, color="purple"
    )  # boxplot will be created and a triangle will indicate the mean value of the column
    sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2, bins=bins
    ) if bins else sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2
    )  # For histogram
    ax_hist2.axvline(
        data[feature].mean(), color="red", linestyle="--"
    )  # Add mean to the histogram
    ax_hist2.axvline(
        data[feature].median(), color="black", linestyle="-"
    )  # Add median to the histogram


# function to create labeled barplots

def labeled_barplot(data, feature,xlabel,ylabel, perc=True, n=None):
    """
    Barplot with percentage at the top

    data: dataframe
    feature: dataframe column
    perc: whether to display percentages instead of count (default is False)
    n: displays the top n category levels (default is None, i.e., display all levels)
    """

    total = len(data[feature])  # length of the column
    count = data[feature].nunique()

    if n is None:
        if count <= 3:
            plt.figure(figsize=(count + 4, count + 4))
        elif count>3 and count <= 15:
            plt.figure(figsize=(count + 2, count + 3 ))    
        else:
            plt.figure(figsize=(count + 200 , count + 300))
    else:
        if n <= 15:
            plt.figure(figsize=(n + 2, count))
        else:
            plt.figure(figsize=(n + 20, count))

            
    
    if count<=3:
        plt.xticks(rotation=65,ha='right', fontsize=count*6,fontweight='bold')# Adjust the fonts of x-axis ticks
        plt.yticks(fontsize=count*4,fontweight='bold')
    elif count>3 and count<=5:        
        plt.xticks(rotation=65,h='right', fontsize=count*4,fontweight='bold')# Adjust the fonts of x-axis ticks
        plt.yticks(fontsize=count*2.5,fontweight='bold')  # Adjust the fontsize for y-axis ticks as needed
    elif count>5 and count<=12:
        plt.xticks(rotation=65,ha='right', fontsize=count*2,fontweight='bold')# Adjust the fonts of x-axis ticks
        plt.yticks(fontsize=count*2.5,fontweight='bold')  # Adjust the fontsize for y-axis ticks as needed
    else:
        plt.xticks(rotation=65,ha='right', fontsize=count*1,fontweight='bold')# Adjust the fonts of x-axis ticks
        plt.yticks(fontsize=count*1.3,fontweight='bold')  # Adjust the fontsize for y-axis ticks as needed
    ax = sns.countplot(
        data=data,
        x=feature,
        palette="Set1",
        order=data[feature].value_counts().index[:n],
    )
    ax.set_xlabel(xlabel)
    if count<=3:
        plt.xlabel(xlabel,rotation=0, fontsize=count*6)# Adjust the fonts of x-axis label
    elif count>3 and count<=5:        
        plt.xlabel(xlabel,rotation=0, fontsize=count*4)# Adjust the fonts of x-axis label
    elif count>5 and count<=12:
        plt.xlabel(xlabel,rotation=0, fontsize=count*2)# Adjust the fonts of x-axis label
    else:
        plt.xlabel(xlabel,rotation=0, fontsize=count*1.2)# Adjust the fonts of x-axis label
    ax.set_ylabel(ylabel)
    if count<=3:
        plt.ylabel(ylabel,rotation=90, fontsize=count*6)# Adjust the fonts of y-axis label
    elif count>3 and count<=5:        
        plt.ylabel(ylabel,rotation=90, fontsize=count*4)# Adjust the fonts of y-axis label
    elif count>5 and count<=12:
        plt.ylabel(ylabel,rotation=90, fontsize=count*2)# Adjust the fonts of y-axis label
    else:
        plt.ylabel(ylabel,rotation=90, fontsize=count*1.2)# Adjust the fonts of y-axis label

    for patch in ax.patches:
        if perc == True:
            label = "{:.1f}%".format(
                100 * patch.get_height() / total
            )  # percentage of each class of the category
        else:
            label = patch.get_height()  # count of each level of the category

        x = patch.get_x() + patch.get_width() / 2  # width of the plot
        y = patch.get_height()  # height of the plot

        if count<= 15:  # Condition for less than or equal to 15 bars
            ax.annotate(
                label,
                (x, y),
                ha="center",
                va="center",
                size=16,
                xytext=(0, 9),
                textcoords="offset points",
            )  # annotate the percentage
        elif count > 15 and count <= 25:  # Condition for 15 to 25 bars
            ax.annotate(
                label,
                (x, y),
                ha="center",
                va="center",
                size=20,
                xytext=(0, 9),
                textcoords="offset points",
            )  # annotate the percentage
        else:  # Condition for more than 25 bars
            ax.annotate(
                label,
                (x, y),
                ha="center",
                va="center",
                size=35,
                xytext=(0, 9),
                textcoords="offset points",
            )  # annotate the percentage

    plt.show()  # show the plot


histogram_boxplot(df,'Current Price') # Writing the code to create histogram_boxplot for 'Current Price'


histogram_boxplot(df,'Price Change') # Writing the code to create histogram_boxplot for 'Price Change'


histogram_boxplot(df,'Volatility') # Writing the code to create histogram_boxplot for 'Volatility'


histogram_boxplot(df,'ROE') # Writing the code to create histogram_boxplot for 'ROE'


histogram_boxplot(df,'Cash Ratio') # Writing the code to create histogram_boxplot for 'Cash Ratio'


histogram_boxplot(df,'Net Cash Flow') # Writing the code to create histogram_boxplot for 'Net Cash Flow'


histogram_boxplot(df,'Net Income') # Writing the code to create histogram_boxplot for 'Net Income'


histogram_boxplot(df,'Earnings Per Share') # Writing the code to create histogram_boxplot for 'Earnings Per Share'


histogram_boxplot(df,'Estimated Shares Outstanding') # Writing the code to create histogram_boxplot for 'Estimated Shares Outstanding'


histogram_boxplot(df,'P/E Ratio') # Writing the code to create histogram_boxplot for 'P/E Ratio'


histogram_boxplot(df,'P/B Ratio') # Writing the code to create histogram_boxplot for 'P/B Ratio'


labeled_barplot(df,'GICS Sector',xlabel='GICS Sector',ylabel='Count')# Writing the code to create labeled_barplot for 'GICS Sector'


#display the five sectors with the most number of stocks
df["GICS Sector"].value_counts().head(n=5)

Industrials               53
Financials                49
Consumer Discretionary    40
Health Care               40
Information Technology    33
Name: GICS Sector, dtype: int64


labeled_barplot(df,'GICS Sub Industry',xlabel='GICS Sub Industry',ylabel='Count') # Writing the code to create labeled_barplot for 'GICS Sub Industry'


#display the five sub industries with the most number of stocks
df['GICS Sub Industry'].value_counts().head(n=5)

Oil & Gas Exploration & Production    16
REITs                                 14
Industrial Conglomerates              14
Internet Software & Services          12
Electric Utilities                    12
Name: GICS Sub Industry, dtype: int64


# correlation check
plt.figure(figsize=(15, 7))
sns.heatmap(
    df.corr(), annot=True, vmin=-1, vmax=1, fmt=".2f", cmap="Spectral"
)
plt.show()


plt.figure(figsize=(15,8))
sns.barplot(data=df, x='GICS Sector', y='Price Change', ci=False)  #Writing the code to choose the right variables
plt.xticks(rotation=90)
plt.show()


plt.figure(figsize=(15,8))
sns.barplot(data=df, x='Cash Ratio', y='GICS Sector', ci=False)  ## Writing the code to choose the right variables
plt.xticks(rotation=90)
plt.show()


plt.figure(figsize=(15,8))
sns.barplot(data=df, x='P/E Ratio', y='GICS Sector', ci=False)  ## Writing the code to choose the right variables
plt.xticks(rotation=90)
plt.show()


plt.figure(figsize=(15,8))
sns.barplot(data=df, x='Volatility', y='GICS Sector', ci=False)  ## Writing the code to choose the right variables
plt.xticks(rotation=90)
plt.show()


plt.figure(figsize=(15, 12))

numeric_columns = df.select_dtypes(include=np.number).columns.tolist()

for i, variable in enumerate(numeric_columns):
    plt.subplot(3, 4, i + 1)
    plt.boxplot(df[variable], whis=1.5)
    plt.tight_layout()
    plt.title(variable)

plt.show()


# scaling the data before clustering
scaler = StandardScaler()
subset =df[numeric_columns].copy()  ## Writing the code to scale the data
subset_scaled = scaler.fit_transform(subset)


# creating a dataframe of the scaled data
subset_scaled_df = pd.DataFrame(subset_scaled, columns=subset.columns)


k_means_df=subset_scaled_df.copy()


#print average distortions for range of kmeans models fitted to scaled dataset
clusters = range(1, 15)
meanDistortions = []

for k in clusters:
    model = KMeans(n_clusters=k, random_state=1)
    model.fit(subset_scaled_df)
    prediction = model.predict(k_means_df)
    distortion = (
        sum(np.min(cdist(k_means_df, model.cluster_centers_, "euclidean"), axis=1))
        / k_means_df.shape[0]
    )

    meanDistortions.append(distortion)

    print("Number of Clusters:", k, "\tAverage Distortion:", distortion)

plt.plot(clusters, meanDistortions, "bx-")
plt.xlabel("k")
plt.ylabel("Average Distortion")
plt.title("Selecting k with the Elbow Method", fontsize=20)
plt.show()

Number of Clusters: 1 	Average Distortion: 2.5425069919221697
Number of Clusters: 2 	Average Distortion: 2.382318498894466
Number of Clusters: 3 	Average Distortion: 2.2692367155390745
Number of Clusters: 4 	Average Distortion: 2.175554082632614
Number of Clusters: 5 	Average Distortion: 2.136852169310594
Number of Clusters: 6 	Average Distortion: 2.0713095032364506
Number of Clusters: 7 	Average Distortion: 1.9775989987671811
Number of Clusters: 8 	Average Distortion: 1.9652603012304684
Number of Clusters: 9 	Average Distortion: 1.9311613101229932
Number of Clusters: 10 	Average Distortion: 1.8502089919483546
Number of Clusters: 11 	Average Distortion: 1.824801480357901
Number of Clusters: 12 	Average Distortion: 1.7525839358924042
Number of Clusters: 13 	Average Distortion: 1.722563828836958
Number of Clusters: 14 	Average Distortion: 1.6973594266503265


#fit KMeans model and use visualizaer to indicate optimal K value
model = KMeans(random_state=1)
visualizer = KElbowVisualizer(model, k=(1, 15), timings=True)
visualizer.fit(k_means_df)  # fit the data to the visualizer
visualizer.show()  # finalize and render figure

<AxesSubplot:title={'center':'Distortion Score Elbow for KMeans Clustering'}, xlabel='k', ylabel='distortion score'>


#fit KMeans model and provide silhouette scores for range of k clusters
sil_score = []
cluster_list = range(2, 15)
for n_clusters in cluster_list:
    clusterer = KMeans(n_clusters=n_clusters, random_state=1)
    preds = clusterer.fit_predict((subset_scaled_df))
    score = silhouette_score(subset_scaled_df, preds)
    sil_score.append(score)
    print("For n_clusters = {}, the silhouette score is {})".format(n_clusters, score))

#show scores in line graph
plt.plot(cluster_list, sil_score)
plt.show()

For n_clusters = 2, the silhouette score is 0.43969639509980457)
For n_clusters = 3, the silhouette score is 0.4644405674779404)
For n_clusters = 4, the silhouette score is 0.45434371948348606)
For n_clusters = 5, the silhouette score is 0.43169988466492354)
For n_clusters = 6, the silhouette score is 0.36698524210914063)
For n_clusters = 7, the silhouette score is 0.10587349598463779)
For n_clusters = 8, the silhouette score is 0.40021596082234684)
For n_clusters = 9, the silhouette score is 0.4051539505522535)
For n_clusters = 10, the silhouette score is 0.1147132918355368)
For n_clusters = 11, the silhouette score is 0.16268826704862685)
For n_clusters = 12, the silhouette score is 0.12883008320005324)
For n_clusters = 13, the silhouette score is 0.1744804713048242)
For n_clusters = 14, the silhouette score is 0.17597674098501367)


#fit KMeans model and use visualizaer to indicate optimal K value
model = KMeans(random_state=1)
visualizer = KElbowVisualizer(model, k=(2, 15), metric="silhouette", timings=True)
visualizer.fit(subset_scaled_df)  # fit the data to the visualizer
visualizer.show()  # finalize and render figure
plt.show()


#find optimal no. of clusters with silhouette coefficients
visualizer = SilhouetteVisualizer(KMeans(5, random_state=1))
visualizer.fit(subset_scaled_df)
visualizer.show()
plt.show()


#create kmeans cluster model
kmeans = KMeans(n_clusters=5, random_state=1)

#fit model to scaled dataset
kmeans.fit(subset_scaled_df)

KMeans(n_clusters=5, random_state=1)

KMeans(n_clusters=5, random_state=1)


# creating a copy of the original data
df1 = df.copy()

# adding kmeans cluster labels to the original and scaled dataframes
k_means_df["KM_segments"] = kmeans.labels_
df1["KM_segments"] = kmeans.labels_


#group dataset by kmeans cluster labels
km_cluster_profile = df1.groupby("KM_segments").mean()

#add counts for number of stocks in each cluster
km_cluster_profile["Count"] = (
    df1.groupby("KM_segments")["Security"].count().values
)


km_cluster_profile.style.highlight_max(color="lightgreen", axis=0)


# print the names of the companies in each cluster
for cl in df1["KM_segments"].unique():
    print("In cluster {}, the following companies are present:".format(cl))
    print(df1[df1["KM_segments"] == cl]["Security"].unique().to_list())
    print()

In cluster 0, the following companies are present:
['American Airlines Group', 'AbbVie', 'Abbott Laboratories', 'Adobe Systems Inc', 'Archer-Daniels-Midland Co', 'Ameren Corp', 'American Electric Power', 'AFLAC Inc', 'American International Group, Inc.', 'Apartment Investment & Mgmt', 'Assurant Inc', 'Arthur J. Gallagher & Co.', 'Akamai Technologies Inc', 'Albemarle Corp', 'Alaska Air Group Inc', 'Allstate Corp', 'Applied Materials Inc', 'AMETEK Inc', 'Affiliated Managers Group Inc', 'Amgen Inc', 'Ameriprise Financial', 'American Tower Corp A', 'AutoNation Inc', 'Anthem Inc.', 'Aon plc', 'Amphenol Corp', 'Arconic Inc', 'Activision Blizzard', 'AvalonBay Communities, Inc.', 'Broadcom', 'American Water Works Company Inc', 'American Express Co', 'Boeing Company', 'Baxter International Inc.', 'BB&T Corporation', 'Bard (C.R.) Inc.', 'BIOGEN IDEC Inc.', 'The Bank of New York Mellon Corp.', 'Ball Corp', 'Bristol-Myers Squibb', 'Boston Scientific', 'BorgWarner', 'Boston Properties', 'Caterpillar Inc.', 'Chubb Limited', 'CBRE Group', 'Crown Castle International Corp.', 'Carnival Corp.', 'Celgene Corp.', 'CF Industries Holdings Inc', 'Citizens Financial Group', 'Church & Dwight', 'C. H. Robinson Worldwide', 'CIGNA Corp.', 'Cincinnati Financial', 'Comerica Inc.', 'CME Group Inc.', 'Cummins Inc.', 'CMS Energy', 'Centene Corporation', 'CenterPoint Energy', 'Capital One Financial', 'The Cooper Companies', 'CSX Corp.', 'CenturyLink Inc', 'Cognizant Technology Solutions', 'Citrix Systems', 'CVS Health', 'Chevron Corp.', 'Dominion Resources', 'Delta Air Lines', 'Du Pont (E.I.)', 'Deere & Co.', 'Discover Financial Services', 'Quest Diagnostics', 'Danaher Corp.', 'The Walt Disney Company', 'Discovery Communications-A', 'Discovery Communications-C', 'Delphi Automotive', 'Digital Realty Trust', 'Dun & Bradstreet', 'Dover Corp.', 'Dr Pepper Snapple Group', 'Duke Energy', 'DaVita Inc.', 'eBay Inc.', 'Ecolab Inc.', 'Consolidated Edison', 'Equifax Inc.', "Edison Int'l", 'Eastman Chemical', 'Equity Residential', 'Eversource Energy', 'Essex Property Trust, Inc.', 'E*Trade', 'Eaton Corporation', 'Entergy Corp.', 'Edwards Lifesciences', 'Exelon Corp.', "Expeditors Int'l", 'Expedia Inc.', 'Extra Space Storage', 'Fastenal Co', 'Fortune Brands Home & Security', 'FirstEnergy Corp', 'Fidelity National Information Services', 'Fiserv Inc', 'FLIR Systems', 'Fluor Corp.', 'Flowserve Corporation', 'FMC Corporation', 'Federal Realty Investment Trust', 'First Solar Inc', 'Frontier Communications', 'General Dynamics', 'General Growth Properties Inc.', 'Corning Inc.', 'General Motors', 'Genuine Parts', 'Garmin Ltd.', 'Goodyear Tire & Rubber', 'Grainger (W.W.) Inc.', 'Hasbro Inc.', 'Huntington Bancshares', 'HCA Holdings', 'Welltower Inc.', 'HCP Inc.', 'Hartford Financial Svc.Gp.', 'Harley-Davidson', "Honeywell Int'l Inc.", 'HP Inc.', 'Hormel Foods Corp.', 'Henry Schein', 'Host Hotels & Resorts', 'The Hershey Company', 'Humana Inc.', 'International Business Machines', 'IDEXX Laboratories', 'Intl Flavors & Fragrances', 'International Paper', 'Interpublic Group', 'Iron Mountain Incorporated', 'Illinois Tool Works', 'Invesco Ltd.', 'J. B. Hunt Transport Services', 'Jacobs Engineering Group', 'Juniper Networks', 'Kimco Realty', 'Kansas City Southern', 'Leggett & Platt', 'Lennar Corp.', 'Laboratory Corp. of America Holding', 'LKQ Corporation', 'L-3 Communications Holdings', 'Lilly (Eli) & Co.', 'Lockheed Martin Corp.', 'Alliant Energy Corp', 'Leucadia National Corp.', 'Southwest Airlines', 'Level 3 Communications', 'LyondellBasell', 'Mastercard Inc.', 'Mid-America Apartments', 'Macerich', "Marriott Int'l.", 'Masco Corp.', 'Mattel Inc.', "McDonald's Corp.", "Moody's Corp", 'Mondelez International', 'MetLife Inc.', 'Mohawk Industries', 'Mead Johnson', 'McCormick & Co.', 'Martin Marietta Materials', 'Marsh & McLennan', '3M Company', 'Monster Beverage', 'Altria Group Inc', 'Marathon Petroleum', 'M&T Bank Corp.', 'Mettler Toledo', 'Mylan N.V.', 'Navient', 'NASDAQ OMX Group', 'NextEra Energy', 'Newmont Mining Corp. (Hldg. Co.)', 'Nielsen Holdings', 'Norfolk Southern Corp.', 'Northern Trust Corp.', 'Nucor Corp.', 'Newell Brands', 'Realty Income Corporation', 'Omnicom Group', "O'Reilly Automotive", "People's United Financial", 'Pitney-Bowes', 'PACCAR Inc.', 'PG&E Corp.', 'Public Serv. Enterprise Inc.', 'PepsiCo Inc.', 'Principal Financial Group', 'Procter & Gamble', 'Progressive Corp.', 'Pulte Homes Inc.', 'Philip Morris International', 'PNC Financial Services', 'Pentair Ltd.', 'Pinnacle West Capital', 'PPG Industries', 'PPL Corp.', 'Prudential Financial', 'Phillips 66', 'Praxair Inc.', 'PayPal', 'Ryder System', 'Royal Caribbean Cruises Ltd', 'Robert Half International', 'Roper Industries', 'Republic Services Inc', 'SCANA Corp', 'Charles Schwab Corporation', 'Sealed Air', 'Sherwin-Williams', 'SL Green Realty', 'Scripps Networks Interactive Inc.', 'Southern Co.', 'Simon Property Group Inc', 'Stericycle Inc', 'Sempra Energy', 'SunTrust Banks', 'State Street Corp.', 'Skyworks Solutions', 'Synchrony Financial', 'Stryker Corp.', 'Molson Coors Brewing Company', 'Tegna, Inc.', 'Torchmark Corp.', 'Thermo Fisher Scientific', 'TripAdvisor', 'The Travelers Companies Inc.', 'Tractor Supply Company', 'Tyson Foods', 'Tesoro Petroleum Co.', 'Total System Services', 'Texas Instruments', 'Under Armour', 'United Continental Holdings', 'UDR Inc', 'Universal Health Services, Inc.', 'United Health Group Inc.', 'Unum Group', 'Union Pacific', 'United Parcel Service', 'United Technologies', 'Varian Medical Systems', 'Valero Energy', 'Vulcan Materials', 'Vornado Realty Trust', 'Verisk Analytics', 'Verisign Inc.', 'Vertex Pharmaceuticals Inc', 'Ventas Inc', 'Waters Corporation', 'Wec Energy Group Inc', 'Whirlpool Corp.', 'Waste Management Inc.', 'Western Union Co', 'Weyerhaeuser Corp.', 'Wyndham Worldwide', 'Xcel Energy Inc', 'XL Capital', 'Dentsply Sirona', 'Xerox Corp.', 'Xylem Inc.', 'Yahoo Inc.', 'Yum! Brands Inc', 'Zimmer Biomet Holdings', 'Zions Bancorp', 'Zoetis']

In cluster 1, the following companies are present:
['Analog Devices, Inc.', 'Alexion Pharmaceuticals', 'Anadarko Petroleum Corp', 'Baker Hughes Inc', 'Cabot Oil & Gas', 'Concho Resources', 'Devon Energy Corp.', 'EOG Resources', 'EQT Corporation', 'Freeport-McMoran Cp & Gld', 'Halliburton Co.', 'Hess Corporation', 'Hewlett Packard Enterprise', 'Kinder Morgan', 'The Mosaic Company', 'Marathon Oil Corp.', 'Murphy Oil', 'Noble Energy Inc', 'Netflix Inc.', 'Newfield Exploration Co', 'National Oilwell Varco Inc.', 'ONEOK', 'Occidental Petroleum', 'Quanta Services Inc.', 'Range Resources Corp.', 'Spectra Energy Corp.', 'Southwestern Energy', 'Teradata Corp.', 'Williams Cos.', 'Wynn Resorts Ltd', 'Cimarex Energy']

In cluster 2, the following companies are present:
['Alliance Data Systems', 'Amazon.com Inc', 'Chipotle Mexican Grill', 'Equinix', 'Facebook', 'Intuitive Surgical Inc.', 'Priceline.com Inc', 'Regeneron']

In cluster 3, the following companies are present:
['Allegion', 'Apache Corporation', 'Chesapeake Energy', 'Charter Communications', 'Colgate-Palmolive', 'Kimberly-Clark', 'S&P Global, Inc.']

In cluster 4, the following companies are present:
['Bank of America Corp', 'Citigroup Inc.', 'Ford Motor', 'Gilead Sciences', 'Intel Corp.', 'JPMorgan Chase & Co.', 'Coca Cola Company', 'Merck & Co.', 'Pfizer Inc.', 'AT&T Inc', 'Verizon Communications', 'Wells Fargo', 'Exxon Mobil Corp.']


#print number of stocks within each sector for all of the clusters
for k in range(0,df1['KM_segments'].nunique()):
    print('The number of stocks within each GICS Sector for Cluster '+str(k)+' are:')
    print(df1[df1['KM_segments']==k]['GICS Sector'].value_counts())
    print("   ")

The number of stocks within each GICS Sector for Cluster 0 are:
Industrials                    51
Financials                     44
Consumer Discretionary         34
Health Care                    34
Information Technology         26
Real Estate                    26
Utilities                      24
Materials                      18
Consumer Staples               16
Energy                          5
Telecommunications Services     3
Name: GICS Sector, dtype: int64
   
The number of stocks within each GICS Sector for Cluster 1 are:
Energy                         22
Information Technology          4
Materials                       2
Consumer Discretionary          1
Health Care                     1
Industrials                     1
Consumer Staples                0
Financials                      0
Real Estate                     0
Telecommunications Services     0
Utilities                       0
Name: GICS Sector, dtype: int64
   
The number of stocks within each GICS Sector for Cluster 2 are:
Consumer Discretionary         3
Health Care                    2
Information Technology         2
Real Estate                    1
Consumer Staples               0
Energy                         0
Financials                     0
Industrials                    0
Materials                      0
Telecommunications Services    0
Utilities                      0
Name: GICS Sector, dtype: int64
   
The number of stocks within each GICS Sector for Cluster 3 are:
Consumer Staples               2
Energy                         2
Consumer Discretionary         1
Financials                     1
Industrials                    1
Health Care                    0
Information Technology         0
Materials                      0
Real Estate                    0
Telecommunications Services    0
Utilities                      0
Name: GICS Sector, dtype: int64
   
The number of stocks within each GICS Sector for Cluster 4 are:
Financials                     4
Health Care                    3
Telecommunications Services    2
Consumer Discretionary         1
Consumer Staples               1
Energy                         1
Information Technology         1
Industrials                    0
Materials                      0
Real Estate                    0
Utilities                      0
Name: GICS Sector, dtype: int64


 #show boxplots of numerical variables for each K-Means cluster
fig, axes = plt.subplots(3, 4, figsize=(20, 20))
counter = 0
plt.suptitle("Boxplot of numerical variables for each cluster")

# selecting numerical columns
num_col = df.select_dtypes(include=np.number).columns.tolist()

for i, variable in enumerate(num_col):
    plt.subplot(3, 4, i + 1)
    sns.boxplot(data=df1, x="KM_segments", y=variable)

plt.tight_layout(pad=2.0)


hc_df = subset_scaled_df.copy()


# list of distance metrics
distance_metrics = ["euclidean", "chebyshev", "mahalanobis", "cityblock"]

# list of linkage methods
linkage_methods = ["single", "complete", "average", "weighted"]

high_cophenet_corr = 0
high_dm_lm = [0, 0]

for dm in distance_metrics:
    for lm in linkage_methods:
        Z = linkage(hc_df, metric=dm, method=lm)
        c, coph_dists = cophenet(Z, pdist(hc_df))
        print(
            "Cophenetic correlation for {} distance and {} linkage is {}.".format(
                dm.capitalize(), lm, round(c,4)
            )
        )
        print(" ")
        if high_cophenet_corr < c:
            high_cophenet_corr = c
            high_dm_lm[0] = dm
            high_dm_lm[1] = lm

Cophenetic correlation for Euclidean distance and single linkage is 0.9232.
 
Cophenetic correlation for Euclidean distance and complete linkage is 0.7873.
 
Cophenetic correlation for Euclidean distance and average linkage is 0.9423.
 
Cophenetic correlation for Euclidean distance and weighted linkage is 0.8694.
 
Cophenetic correlation for Chebyshev distance and single linkage is 0.9063.
 
Cophenetic correlation for Chebyshev distance and complete linkage is 0.5989.
 
Cophenetic correlation for Chebyshev distance and average linkage is 0.9338.
 
Cophenetic correlation for Chebyshev distance and weighted linkage is 0.9127.
 
Cophenetic correlation for Mahalanobis distance and single linkage is 0.9259.
 
Cophenetic correlation for Mahalanobis distance and complete linkage is 0.7925.
 
Cophenetic correlation for Mahalanobis distance and average linkage is 0.9247.
 
Cophenetic correlation for Mahalanobis distance and weighted linkage is 0.8708.
 
Cophenetic correlation for Cityblock distance and single linkage is 0.9334.
 
Cophenetic correlation for Cityblock distance and complete linkage is 0.7375.
 
Cophenetic correlation for Cityblock distance and average linkage is 0.9302.
 
Cophenetic correlation for Cityblock distance and weighted linkage is 0.731.


# printing the combination of distance metric and linkage method with the highest cophenetic correlation
print(
    "Highest cophenetic correlation is {}, which is obtained with {} distance and {} linkage.".format(
        round(high_cophenet_corr,4), high_dm_lm[0].capitalize(), high_dm_lm[1]
    )
)

Highest cophenetic correlation is 0.9423, which is obtained with Euclidean distance and average linkage.


# list of linkage methods for euclidean distance metric
linkage_methods = ["single", "complete", "average", "centroid", "ward", "weighted"]

high_cophenet_corr = 0
high_dm_lm = [0, 0]

for lm in linkage_methods:
    Z = linkage(hc_df, metric="euclidean", method=lm)
    c, coph_dists = cophenet(Z, pdist(hc_df))
    print(
            "Cophenetic correlation for Euclidean distance and {} linkage is {}.".format(
                lm, round(c,4)
            )
        )
    print(" ")
    if high_cophenet_corr < c:
        high_cophenet_corr = c
        high_dm_lm[0] = "euclidean"
        high_dm_lm[1] = lm

Cophenetic correlation for Euclidean distance and single linkage is 0.9232.
 
Cophenetic correlation for Euclidean distance and complete linkage is 0.7873.
 
Cophenetic correlation for Euclidean distance and average linkage is 0.9423.
 
Cophenetic correlation for Euclidean distance and centroid linkage is 0.9314.
 
Cophenetic correlation for Euclidean distance and ward linkage is 0.7101.
 
Cophenetic correlation for Euclidean distance and weighted linkage is 0.8694.


# printing the combination of distance metric and linkage method with the highest cophenetic correlation
print(
    "Highest cophenetic correlation is {}, which is obtained with {} linkage.".format(
        round(high_cophenet_corr,4), high_dm_lm[1]
    )
)

Highest cophenetic correlation is 0.9423, which is obtained with average linkage.


# list of linkage methods
linkage_methods = ["single", "complete", "average", "centroid", "ward", "weighted"]

# lists to save results of cophenetic correlation calculation
compare_cols = ["Linkage", "Cophenetic Coefficient"]
compare=[]

# to create a subplot image
fig, axs = plt.subplots(len(linkage_methods), 1, figsize=(15, 30))

# We will enumerate through the list of linkage methods above
# For each linkage method, we will plot the dendrogram and calculate the cophenetic correlation
for i, method in enumerate(linkage_methods):
    Z = linkage(hc_df, metric="euclidean", method=method)

    dendrogram(Z, ax=axs[i])
    axs[i].set_title(f"Dendrogram ({method.capitalize()} Linkage)")

    coph_corr, coph_dist = cophenet(Z, pdist(subset_scaled_df))
    axs[i].annotate(
        f"Cophenetic\nCorrelation\n{coph_corr:0.2f}",
        (0.80, 0.80),
        xycoords="axes fraction",
    )


# create and print a dataframe to compare cophenetic correlations for different linkage methods
df_cc = pd.DataFrame(compare, columns=compare_cols)
df_cc = df_cc.sort_values(by="Cophenetic Coefficient")
df_cc


Z = linkage(hc_df, metric='euclidean', method='average')
c, coph_dists = cophenet(Z , pdist(hc_df))


hierarchy = AgglomerativeClustering(n_clusters=5, affinity='euclidean',  linkage='average')
hierarchy.fit(hc_df)

AgglomerativeClustering(affinity='euclidean', linkage='average', n_clusters=5)

AgglomerativeClustering(affinity='euclidean', linkage='average', n_clusters=5)


df_hierarchy = df.copy()
df_hierarchy['HC_clusters'] = hierarchy.labels_


#group dataset by Hierarchical clusters
cluster_profile_h = df_hierarchy.groupby("HC_clusters").mean()

#add counts for number of stocks in each cluster
cluster_profile_h["Count"] = (
    df_hierarchy.groupby("HC_clusters")["Current Price"].count().values
)

#show dataframe with maximum values for each metric highlighted
cluster_profile_h.style.highlight_max(color="lightgreen", axis=0)


HCmodel = AgglomerativeClustering(n_clusters=5, affinity="euclidean", linkage="ward")
HCmodel.fit(hc_df)

AgglomerativeClustering(affinity='euclidean', n_clusters=5)

AgglomerativeClustering(affinity='euclidean', n_clusters=5)


hc_df["HC_clusters"] = HCmodel.labels_
df_hierarchy["HC_clusters"] = HCmodel.labels_


#group dataset by Hierarchical clusters
cluster_profile_h = df_hierarchy.groupby("HC_clusters").mean()

#add counts for number of stocks in each cluster
cluster_profile_h["Count"] = (
    df_hierarchy.groupby("HC_clusters")["Current Price"].count().values
)

#show dataframe with maximum values for each metric highlighted
cluster_profile_h.style.highlight_max(color="lightgreen", axis=0)


# print the names of the companies in each cluster
for cl in df_hierarchy["HC_clusters"].unique():
    print("In cluster {}, the following companies are present:".format(cl))
    print(df_hierarchy[df_hierarchy["HC_clusters"] == cl]["Security"].unique().to_list())
    print()

In cluster 3, the following companies are present:
['American Airlines Group', 'AbbVie', 'Abbott Laboratories', 'Adobe Systems Inc', 'Analog Devices, Inc.', 'Archer-Daniels-Midland Co', 'Ameren Corp', 'American Electric Power', 'AFLAC Inc', 'American International Group, Inc.', 'Apartment Investment & Mgmt', 'Assurant Inc', 'Arthur J. Gallagher & Co.', 'Akamai Technologies Inc', 'Albemarle Corp', 'Alaska Air Group Inc', 'Allstate Corp', 'Applied Materials Inc', 'AMETEK Inc', 'Affiliated Managers Group Inc', 'Ameriprise Financial', 'American Tower Corp A', 'AutoNation Inc', 'Anthem Inc.', 'Aon plc', 'Amphenol Corp', 'Arconic Inc', 'Activision Blizzard', 'AvalonBay Communities, Inc.', 'Broadcom', 'American Water Works Company Inc', 'American Express Co', 'Boeing Company', 'Baxter International Inc.', 'BB&T Corporation', 'Bard (C.R.) Inc.', 'BIOGEN IDEC Inc.', 'The Bank of New York Mellon Corp.', 'Ball Corp', 'Bristol-Myers Squibb', 'Boston Scientific', 'BorgWarner', 'Boston Properties', 'Caterpillar Inc.', 'Chubb Limited', 'CBRE Group', 'Crown Castle International Corp.', 'Carnival Corp.', 'Celgene Corp.', 'CF Industries Holdings Inc', 'Citizens Financial Group', 'Church & Dwight', 'C. H. Robinson Worldwide', 'CIGNA Corp.', 'Cincinnati Financial', 'Comerica Inc.', 'CME Group Inc.', 'Cummins Inc.', 'CMS Energy', 'Centene Corporation', 'CenterPoint Energy', 'Capital One Financial', 'The Cooper Companies', 'CSX Corp.', 'CenturyLink Inc', 'Cognizant Technology Solutions', 'Citrix Systems', 'CVS Health', 'Chevron Corp.', 'Dominion Resources', 'Delta Air Lines', 'Du Pont (E.I.)', 'Deere & Co.', 'Discover Financial Services', 'Quest Diagnostics', 'Danaher Corp.', 'The Walt Disney Company', 'Discovery Communications-A', 'Discovery Communications-C', 'Delphi Automotive', 'Digital Realty Trust', 'Dun & Bradstreet', 'Dover Corp.', 'Dr Pepper Snapple Group', 'Duke Energy', 'DaVita Inc.', 'eBay Inc.', 'Ecolab Inc.', 'Consolidated Edison', 'Equifax Inc.', "Edison Int'l", 'Eastman Chemical', 'Equity Residential', 'EQT Corporation', 'Eversource Energy', 'Essex Property Trust, Inc.', 'E*Trade', 'Eaton Corporation', 'Entergy Corp.', 'Edwards Lifesciences', 'Exelon Corp.', "Expeditors Int'l", 'Expedia Inc.', 'Extra Space Storage', 'Fastenal Co', 'Fortune Brands Home & Security', 'FirstEnergy Corp', 'Fidelity National Information Services', 'Fiserv Inc', 'FLIR Systems', 'Fluor Corp.', 'Flowserve Corporation', 'FMC Corporation', 'Federal Realty Investment Trust', 'First Solar Inc', 'General Dynamics', 'General Growth Properties Inc.', 'Gilead Sciences', 'Corning Inc.', 'General Motors', 'Genuine Parts', 'Garmin Ltd.', 'Goodyear Tire & Rubber', 'Grainger (W.W.) Inc.', 'Halliburton Co.', 'Hasbro Inc.', 'Huntington Bancshares', 'HCA Holdings', 'Welltower Inc.', 'HCP Inc.', 'Hartford Financial Svc.Gp.', 'Harley-Davidson', "Honeywell Int'l Inc.", 'Hewlett Packard Enterprise', 'HP Inc.', 'Hormel Foods Corp.', 'Henry Schein', 'Host Hotels & Resorts', 'The Hershey Company', 'Humana Inc.', 'International Business Machines', 'IDEXX Laboratories', 'Intl Flavors & Fragrances', 'International Paper', 'Interpublic Group', 'Iron Mountain Incorporated', 'Illinois Tool Works', 'Invesco Ltd.', 'J. B. Hunt Transport Services', 'Jacobs Engineering Group', 'Juniper Networks', 'Kimco Realty', 'Kansas City Southern', 'Leggett & Platt', 'Lennar Corp.', 'Laboratory Corp. of America Holding', 'LKQ Corporation', 'L-3 Communications Holdings', 'Lilly (Eli) & Co.', 'Lockheed Martin Corp.', 'Alliant Energy Corp', 'Leucadia National Corp.', 'Southwest Airlines', 'Level 3 Communications', 'LyondellBasell', 'Mastercard Inc.', 'Mid-America Apartments', 'Macerich', "Marriott Int'l.", 'Masco Corp.', 'Mattel Inc.', "McDonald's Corp.", "Moody's Corp", 'Mondelez International', 'MetLife Inc.', 'Mohawk Industries', 'Mead Johnson', 'McCormick & Co.', 'Martin Marietta Materials', 'Marsh & McLennan', '3M Company', 'Altria Group Inc', 'The Mosaic Company', 'Marathon Petroleum', 'Merck & Co.', 'M&T Bank Corp.', 'Mettler Toledo', 'Mylan N.V.', 'Navient', 'NASDAQ OMX Group', 'NextEra Energy', 'Newmont Mining Corp. (Hldg. Co.)', 'Nielsen Holdings', 'Norfolk Southern Corp.', 'Northern Trust Corp.', 'Nucor Corp.', 'Newell Brands', 'Realty Income Corporation', 'Omnicom Group', "O'Reilly Automotive", "People's United Financial", 'Pitney-Bowes', 'PACCAR Inc.', 'PG&E Corp.', 'Public Serv. Enterprise Inc.', 'PepsiCo Inc.', 'Principal Financial Group', 'Procter & Gamble', 'Progressive Corp.', 'Pulte Homes Inc.', 'Philip Morris International', 'PNC Financial Services', 'Pentair Ltd.', 'Pinnacle West Capital', 'PPG Industries', 'PPL Corp.', 'Prudential Financial', 'Phillips 66', 'Quanta Services Inc.', 'Praxair Inc.', 'PayPal', 'Ryder System', 'Royal Caribbean Cruises Ltd', 'Robert Half International', 'Roper Industries', 'Republic Services Inc', 'SCANA Corp', 'Charles Schwab Corporation', 'Sealed Air', 'Sherwin-Williams', 'SL Green Realty', 'Scripps Networks Interactive Inc.', 'Southern Co.', 'Simon Property Group Inc', 'Stericycle Inc', 'Sempra Energy', 'SunTrust Banks', 'State Street Corp.', 'Skyworks Solutions', 'Synchrony Financial', 'Stryker Corp.', 'Molson Coors Brewing Company', 'Tegna, Inc.', 'Torchmark Corp.', 'Thermo Fisher Scientific', 'TripAdvisor', 'The Travelers Companies Inc.', 'Tractor Supply Company', 'Tyson Foods', 'Tesoro Petroleum Co.', 'Total System Services', 'Texas Instruments', 'Under Armour', 'United Continental Holdings', 'UDR Inc', 'Universal Health Services, Inc.', 'United Health Group Inc.', 'Unum Group', 'Union Pacific', 'United Parcel Service', 'United Technologies', 'Varian Medical Systems', 'Valero Energy', 'Vulcan Materials', 'Vornado Realty Trust', 'Verisk Analytics', 'Verisign Inc.', 'Vertex Pharmaceuticals Inc', 'Ventas Inc', 'Wec Energy Group Inc', 'Whirlpool Corp.', 'Waste Management Inc.', 'Western Union Co', 'Weyerhaeuser Corp.', 'Wyndham Worldwide', 'Wynn Resorts Ltd', 'Xcel Energy Inc', 'XL Capital', 'Dentsply Sirona', 'Xerox Corp.', 'Xylem Inc.', 'Yum! Brands Inc', 'Zimmer Biomet Holdings', 'Zions Bancorp', 'Zoetis']

In cluster 0, the following companies are present:
['Alliance Data Systems', 'Alexion Pharmaceuticals', 'Amgen Inc', 'Amazon.com Inc', 'Chipotle Mexican Grill', 'Equinix', 'Facebook', 'Frontier Communications', 'Intuitive Surgical Inc.', 'Monster Beverage', 'Netflix Inc.', 'Priceline.com Inc', 'Regeneron', 'Waters Corporation', 'Yahoo Inc.']

In cluster 1, the following companies are present:
['Allegion', 'Apache Corporation', 'Chesapeake Energy', 'Charter Communications', 'Colgate-Palmolive', 'Kimberly-Clark', 'S&P Global, Inc.']

In cluster 4, the following companies are present:
['Anadarko Petroleum Corp', 'Baker Hughes Inc', 'Cabot Oil & Gas', 'Concho Resources', 'Devon Energy Corp.', 'EOG Resources', 'Freeport-McMoran Cp & Gld', 'Hess Corporation', 'Kinder Morgan', 'Marathon Oil Corp.', 'Murphy Oil', 'Noble Energy Inc', 'Newfield Exploration Co', 'National Oilwell Varco Inc.', 'ONEOK', 'Occidental Petroleum', 'Range Resources Corp.', 'Spectra Energy Corp.', 'Southwestern Energy', 'Teradata Corp.', 'Williams Cos.', 'Cimarex Energy']

In cluster 2, the following companies are present:
['Bank of America Corp', 'Citigroup Inc.', 'Ford Motor', 'Intel Corp.', 'JPMorgan Chase & Co.', 'Coca Cola Company', 'Pfizer Inc.', 'AT&T Inc', 'Verizon Communications', 'Wells Fargo', 'Exxon Mobil Corp.']


# print the number of stocks in each GICS sector for each cluster
for k in range(0,df_hierarchy['HC_clusters'].nunique()):
    print('The number of stocks within each GICS Sector for Cluster '+str(k)+' are:')
    print(df_hierarchy[df_hierarchy['HC_clusters']==k]['GICS Sector'].value_counts())
    print("   ")

The number of stocks within each GICS Sector for Cluster 0 are:
Health Care                    5
Information Technology         4
Consumer Discretionary         3
Consumer Staples               1
Real Estate                    1
Telecommunications Services    1
Energy                         0
Financials                     0
Industrials                    0
Materials                      0
Utilities                      0
Name: GICS Sector, dtype: int64
   
The number of stocks within each GICS Sector for Cluster 1 are:
Consumer Staples               2
Energy                         2
Consumer Discretionary         1
Financials                     1
Industrials                    1
Health Care                    0
Information Technology         0
Materials                      0
Real Estate                    0
Telecommunications Services    0
Utilities                      0
Name: GICS Sector, dtype: int64
   
The number of stocks within each GICS Sector for Cluster 2 are:
Financials                     4
Telecommunications Services    2
Consumer Discretionary         1
Consumer Staples               1
Energy                         1
Health Care                    1
Information Technology         1
Industrials                    0
Materials                      0
Real Estate                    0
Utilities                      0
Name: GICS Sector, dtype: int64
   
The number of stocks within each GICS Sector for Cluster 3 are:
Industrials                    52
Financials                     44
Consumer Discretionary         35
Health Care                    34
Information Technology         27
Real Estate                    26
Utilities                      24
Materials                      19
Consumer Staples               15
Energy                          7
Telecommunications Services     2
Name: GICS Sector, dtype: int64
   
The number of stocks within each GICS Sector for Cluster 4 are:
Energy                         20
Information Technology          1
Materials                       1
Consumer Discretionary          0
Consumer Staples                0
Financials                      0
Health Care                     0
Industrials                     0
Real Estate                     0
Telecommunications Services     0
Utilities                       0
Name: GICS Sector, dtype: int64


# show boxplots of numerical variables for each Hierarchical cluster
fig, axes = plt.subplots(3, 4, figsize=(20, 20))
counter = 0

for ii in range(3):
    for jj in range(4):
        if counter < 11:
            sns.boxplot(
                ax=axes[ii][jj],
                data=df_hierarchy,
                y=df_hierarchy.columns[3+counter],
                x="HC_clusters",
                palette="viridis"
            )
            counter = counter + 1

fig.tight_layout(pad=3.0)

	Ticker Symbol	Security	GICS Sector	GICS Sub Industry	Current Price	Price Change	Volatility	ROE	Cash Ratio	Net Cash Flow	Net Income	Earnings Per Share	Estimated Shares Outstanding	P/E Ratio	P/B Ratio
102	DVN	Devon Energy Corp.	Energy	Oil & Gas Exploration & Production	32.00	-15.48	2.92	205	70	830000000	-14454000000	-35.55	406582278.50	93.09	1.79
125	FB	Facebook	Information Technology	Internet Software & Services	104.66	16.22	1.32	8	958	592000000	3669000000	1.31	2800763359.00	79.89	5.88
11	AIV	Apartment Investment & Mgmt	Real Estate	REITs	40.03	7.58	1.16	15	47	21818000	248710000	1.52	163625000.00	26.34	-1.27
248	PG	Procter & Gamble	Consumer Staples	Personal Products	79.41	10.66	0.81	17	129	160383000	636056000	3.28	491391569.00	24.07	-2.26
238	OXY	Occidental Petroleum	Energy	Oil & Gas Exploration & Production	67.61	0.87	1.59	32	64	-588000000	-7829000000	-10.23	765298142.70	93.09	3.35
336	YUM	Yum! Brands Inc	Consumer Discretionary	Restaurants	52.52	-8.70	1.48	142	27	159000000	1293000000	2.97	435353535.40	17.68	-3.84
112	EQT	EQT Corporation	Energy	Oil & Gas Exploration & Production	52.13	-21.25	2.36	2	201	523803000	85171000	0.56	152091071.40	93.09	9.57
147	HAL	Halliburton Co.	Energy	Oil & Gas Equipment & Services	34.04	-5.10	1.97	4	189	7786000000	-671000000	-0.79	849367088.60	93.09	17.35
89	DFS	Discover Financial Services	Financials	Consumer Finance	53.62	3.65	1.16	20	99	2288000000	2297000000	5.14	446887159.50	10.43	-0.38
173	IVZ	Invesco Ltd.	Financials	Asset Management & Custody Banks	33.48	7.07	1.58	12	67	412000000	968100000	2.26	428362831.90	14.81	4.22

	count	unique	top	freq	mean	std	min	25%	50%	75%	max
Security	340	340	3M Company	1	NaN	NaN	NaN	NaN	NaN	NaN	NaN
GICS Sector	340	11	Industrials	53	NaN	NaN	NaN	NaN	NaN	NaN	NaN
GICS Sub Industry	340	104	Oil & Gas Exploration & Production	16	NaN	NaN	NaN	NaN	NaN	NaN	NaN
Current Price	340.00	NaN	NaN	NaN	80.86	98.06	4.50	38.55	59.70	92.88	1274.95
Price Change	340.00	NaN	NaN	NaN	4.08	12.01	-47.13	-0.94	4.82	10.70	55.05
Volatility	340.00	NaN	NaN	NaN	1.53	0.59	0.73	1.13	1.39	1.70	4.58
ROE	340.00	NaN	NaN	NaN	39.60	96.55	1.00	9.75	15.00	27.00	917.00
Cash Ratio	340.00	NaN	NaN	NaN	70.02	90.42	0.00	18.00	47.00	99.00	958.00
Net Cash Flow	340.00	NaN	NaN	NaN	55537620.59	1946365312.18	-11208000000.00	-193906500.00	2098000.00	169810750.00	20764000000.00
Net Income	340.00	NaN	NaN	NaN	1494384602.94	3940150279.33	-23528000000.00	352301250.00	707336000.00	1899000000.00	24442000000.00
Earnings Per Share	340.00	NaN	NaN	NaN	2.78	6.59	-61.20	1.56	2.90	4.62	50.09
Estimated Shares Outstanding	340.00	NaN	NaN	NaN	577028337.75	845849595.42	27672156.86	158848216.10	309675137.80	573117457.32	6159292035.00
P/E Ratio	340.00	NaN	NaN	NaN	32.61	44.35	2.94	15.04	20.82	31.76	528.04
P/B Ratio	340.00	NaN	NaN	NaN	-1.72	13.97	-76.12	-4.35	-1.07	3.92	129.06

	Current Price	Price Change	Volatility	ROE	Cash Ratio	Net Cash Flow	Net Income	Earnings Per Share	Estimated Shares Outstanding	P/E Ratio	P/B Ratio	Count
KM_segments
0	73.642268	5.604746	1.398809	25.071174	65.217082	19568583.629893	1477791896.797153	3.677100	435833507.534555	23.723142	-2.643991	281
1	44.533549	-11.758615	2.773054	48.193548	70.032258	169199225.806452	-2102799709.677419	-5.385484	528269475.759355	96.953796	1.719411	31
2	525.418743	8.811119	1.484593	16.375000	259.000000	258205375.000000	1162641625.000000	12.772500	451665922.790000	113.904340	30.234451	8
3	84.355716	3.854981	1.827670	633.571429	33.571429	-568400000.000000	-4968157142.857142	-10.841429	398169036.442857	42.284541	-11.589502	7
4	48.103077	6.053507	1.163964	27.538462	77.230769	773230769.230769	14114923076.923077	3.958462	3918734987.169230	16.098039	-4.253404	13

	Current Price	Price Change	Volatility	ROE	Cash Ratio	Net Cash Flow	Net Income	Earnings Per Share	Estimated Shares Outstanding	P/E Ratio	P/B Ratio	Count
HC_clusters
0	77.884243	4.105986	1.516865	35.320359	66.775449	-32825817.365269	1535255703.592814	2.903308	559027333.145509	32.437511	-1.781988	334
1	25.640000	11.237908	1.322355	12.500000	130.500000	16755500000.000000	13654000000.000000	3.295000	2791829362.100000	13.649696	1.508484	2
2	24.485001	-13.351992	3.482611	802.000000	51.000000	-1292500000.000000	-19106500000.000000	-41.815000	519573983.250000	60.748608	1.565141	2
3	104.660004	16.224320	1.320606	8.000000	958.000000	592000000.000000	3669000000.000000	1.310000	2800763359.000000	79.893133	5.884467	1
4	1274.949951	3.190527	1.268340	29.000000	184.000000	-1671386000.000000	2551360000.000000	50.090000	50935516.070000	25.453183	-1.052429	1

	Current Price	Price Change	Volatility	ROE	Cash Ratio	Net Cash Flow	Net Income	Earnings Per Share	Estimated Shares Outstanding	P/E Ratio	P/B Ratio	Count
HC_clusters
0	326.198218	10.563242	1.642560	14.400000	309.466667	288850666.666667	864498533.333333	7.785333	544900261.301333	113.095334	19.142151	15
1	84.355716	3.854981	1.827670	633.571429	33.571429	-568400000.000000	-4968157142.857142	-10.841429	398169036.442857	42.284541	-11.589502	7
2	42.848182	6.270446	1.123547	22.727273	71.454545	558636363.636364	14631272727.272728	3.410000	4242572567.290909	15.242169	-4.924615	11
3	72.760400	5.213307	1.427078	25.603509	60.392982	79951512.280702	1538594322.807018	3.655351	446472132.228456	24.722670	-2.647194	285
4	36.440455	-16.073408	2.832884	57.500000	42.409091	-472834090.909091	-3161045227.272727	-8.005000	514367806.201818	85.555682	0.836839	22

Unsupervised Learning: Trade&Ahead¶

Context¶

Objective¶

Data Dictionary¶

Importing necessary libraries and data¶

Data Overview¶

Checking The Shape Of The Dataset¶

Displaying Few Rows Of The Dataset¶

Checking The Datatypes of The Column Of The Dataset¶

Checking For Duplicates And Missing Values¶

Statistical Summary of The Dataset¶

Exploratory Data Analysis (EDA)¶

Univariate Analysis¶

Current Price¶

Price Change¶

Volatitlity¶

ROE¶

Cash Ratio¶

Net Cash Flow¶

Net Income¶

Earnings Per Share¶

Estimated Shares Outstanding¶

P/E Ratio¶

P/B Ratio¶

GICS Sector¶

GICS Sub Industry¶

Bivariate Analysis¶

Let's check the stocks of which economic sector have seen the maximum price increase on average.¶

Cash ratio provides a measure of a company's ability to cover its short-term obligations using only cash and cash equivalents. Let's see how the average cash ratio varies across economic sectors.¶

P/E ratios can help determine the relative value of a company's shares as they signify the amount of money an investor is willing to invest in a single share of a company per dollar of its earnings. Let's see how the P/E ratio varies, on average, across economic sectors.¶

Volatility accounts for the fluctuation in the stock price. A stock with high volatility will witness sharper price changes, making it a riskier investment. Let's see how volatility varies, on average, across economic sectors.¶

Data Preprocessing¶

Outlier Check¶

Scaling¶

K-means Clustering¶

Checking Elbow Plot¶

Let's check the silhouette scores¶

Creating Final Model¶

Cluster Profiling¶

Insights¶

Hierarchical Clustering¶

Computing Cophenetic Correlation¶

Let's explore different linkage methods with Euclidean distance only.¶

Let's view the dendrograms for the different linkage methods with Euclidean distance.¶

Checking Dendograms¶

Creating Model Using sklearn¶

Cluster Profiling¶

Cluster Profiling¶

Insights¶

K-means vs Hierarchical Clustering¶

You compare several things, like:¶

You can also mention any differences or similarities you obtained in the cluster profiles from both the clustering techniques.¶

Actionable Insights and Recommendations¶